Machine learning (ML)-based data flow solutions have made it possible to ingest and process data from a large number of applications at an affordable cost. This not only helps expand the overall scope of threat detection, but also helps significantly accelerate the development and production of threat detection applications. Solutions that offer advanced capabilities like in-memory data transformation and distributed in-memory stateful processing also bolster insider threat detection by enabling faster data quality scoring, cleansing, and enrichment. Recent advances in ML have helped create dynamic models that periodically learn normal baseline behavior and detect anomalies based on both dynamic and static factors such as identities, roles, and excess access permissions; correlated with log and event data. Using ML models on the log and complex event data can help reduce false positives from thousands to tens per day and make the end-to-end process of identifying suspicious behavior automated, accurate, and timely.

So if you've been following my content, you know I've been writing a lot about artificial intelligence. I've shown some of the positive and negative developments in this area, and how we should harness this immensely powerful technology for the commonwealth of man; and not exploit its use for evil, the way we historically have with nuclear weaponry. There are many thinkers and innovators who have been advocating for this. In line with that, today's piece is all about the ethical principles that pundits feel should be programmed into AI, and developed with a clear view in mind moving forward. I've loosely been basing these artificial intelligence articles around an incredible book entitled 2084 written by Professor John Lennox.

Modern software systems emit a tremendous amount of "machine data" (logs, metrics, etc.) that can be crucial to identifying and understanding misbehavior, but the quantity and complexity of this data is outpacing the human ability to do the required analysis and take timely action. For this reason, I think we will see a lot of opportunities to build automated systems that analyze (and even act) on this machine data in order to improve the security, performance, and reliability of economically critical software services. That said, there's also a lot of exciting research around "ML on code": automatically identifying risky pull requests, automated bug localization, intelligent IDE assistance, and so on. Given the well-known challenges of building and operating software systems, there is likely to be plenty of room for improvement across the entire lifecycle. Overall, I think we're heading into a really interesting time for the application of ML techniques to software development, security, and operations.